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ABSTRACT 


-^Conjoint  measurement  theory  is  examined  through  a 
prototype  example  in  which  a  fighter  aircraft  is  subjectively 
rated  on  two  factors.  As  a  first  step,  a  multifactor  ordinal 
scale  is  developed.  This  ordinal  scale  provides  a  meaningful 
measure  of  aircraft  quality.  Interval  scales  of  aircraft 
quality  are  produced  by  the  basic  analysis  of  variance  model 
and  two  conjoint  measurement  methods:  delta  scaling  and  the 
computer  algorithm  MONANOVA.  These  methods  produce  interval 
scales  that  differ  by  constant  factors,  as  guaranteed  by 
the  theorem  for  additive  conjoint  measurement.  The  interval 
scale  does  not  appear  to  be  an  improvement  over  the  ordinal 
scale  in  the  prototype  example.  There  is  no  assurance  that 
a  specific  conjoint  measurement  model  can  be  used  to  "improve 
*  >e  data.  Major  changes  in  the  interval  scales  are  caused 
by  small  perturbations  in  the  rating  matrix. 
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INTRODUCTION 


Subjective  assessments  of  aircraft  quality  and  pilot 
workload  have  been  shown  to  be  practical,  minimally  intrusive, 
and  not  disruptive  of  primary  task  performance  (Eggemeier, 
1980). 

Sheridan  and  Simpson  (1979)  developed  a  prototype 
subjective  rating  method  for  Instrument  Flight  Rules  pilot 
workload  modeled  closely  after  the  Cooper-Harper  (1969)  rating 
system  which  was  developed  to  evaluate  aircraft  handling 
qualities.  Rehmann  et.al.  (1983)  developed  a  technique  by 
which  participants  subjectively  rated  workload  every  60  s 
during  the  task,  using  a  mechanical  device,  to  overcome  bias 
effects  in  subjective  ratings  at  the  conclusion  of  the  task. 
Wierwille  and  Conner  (1983)  evaluated  twenty  workload  measures 
using  a  psychomotor  task  in  a  moving-base  aircraft  simulator. 
They  concluded  that  well-designed  subjective  rating  scales 
were  among  the  best  techniques  for  evaluating  psychomotor 
load. 

Conjoint  measurement  theory,  which  might  best  be 
described  as  a  general  modeling  and  scaling  approach,  has 
been  used  to  develop  multidimensional  subjective  measures 
of  workload.  Reid  et.al.  (1981)  used  conjoint  measurement 
theory  to  construct  interval  level  workload  scales  from  ordinal 
rankings  of  combinations  of  levels  on  three  contributory 
workload  scales  from  ordinal  ranking  of  combinations  of  levels 


on  three  contributory  dimensions.  The  dimensions  were  time 
load,  mental  effort  load,  and  psychological  stress  load. 

They  concluded  that  their  subjective  assessment  technique 
was  sensitive  to  differences  in  workload  on  a  critically 
unstable  tracking  task. 

Donnell  and  O'Connor  (1978)  and  Donnell  (1979)  applied 
conjoint  measurement  theory  to  fighter  aircraft  to  develop 
an  interval  scale  measure  of  aircraft  system  operability. 

System  operability  was  assumed  to  represent  a  combination 
of  a  number  of  factors,  including  the  amount  of  workload 
required  by  the  operator  in  performance  of  a  task,  and  the 
degree  of  subsystem  technical  effectiveness  demonstrated 
in  accomplishment  of  the  task.  In  order  to  develop  an  interval 
scale  of  systems  operability,  Donnell  and  O'Connor  (1978) 
developed  separate  four-point  ordinal  rating  scales  for  pilot 
workload  and  system  technical  effectiveness.  Using  conjoint 
measurement  and  related  scaling  procedures,  independent  ratings 
by  pilots  on  each  of  these  two  dimensions  were  eventually 
combined  and  converted  into  an  interval  scale  of  systems 
operability.  Donnell  (1979)  subsequently  modified  the  pilot 
workload  and  technical  effectiveness  scales  and  applied  them 
to  a  different  aircraft.  An  interval  scale  of  systems  operability 
was  also  derived  in  this  application  with  use  of  the  conjoint 
measurement  technique. 

Whereas  the  interval  scale  produced  by  conjoint  measurement 
theory  may  be  sensitive  to  changes  in  workload  or  changes 


in  aircraft  quality,  it  has  not  been  demonstrated  that  the 


derived  interval  scale  is  an  improvement  over  the  ordinal 


scale.  Furthermore,  the  ordinal  rating  scale  itself,  upon 


which  inferences  and  scale  development  is  made,  may  not  be 


sufficiently  precise. 


v*.  •-  < 


OBJECTIVES 


The  primary  objective  of  this  research  is  to  investigate 
whether  conjoint  measurement  theory  can  be  effectively  used 
to  improve  the  quantification  of  subjective  ratings  of  aircraft 
quality,  pilot  workload,  etc.  In  particular,  the  interval 
scale  produced  by  conjoint  measurement  theory  will  be  examined 
to  determine  if  it  is  a  true  improvement  over  an  ordinal 
scale.  The  structure  of  conjoint  measurement  theory  and 
the  effectiveness  of  its  representations  are  examined  through 
a  prototype  example  of  aircraft  quality. 


CONJOINT  MEASUREMENT  THEORY 


Conjoint  measurement  theory  is  developed  in  the  following 
primary  references:  Shepard  {1962a,  1962b),  Kruskal  (1964a, 
1964b),  Luce  and  Tukey  (1964),  Kratz  and  Tversky  (1971), 
and  Baird  and  Noma  (1978).  Parts  of  the  following  description 
of  conjoint  measurement  theory  and  associated  algorithms 
are  based  upon  Nygren  (1981). 

Conjoint  measurement  can  be  defined  as  the  procedure 
whereby  we  specify  for  a  given  combination  rule,  the  conditions 
under  which  there  exist  measurement  scales  for  the  dependent 
and  independent  variables,  such  that  the  order  of  the  joint 
effects  of  the  independent  variables  in  the  data  are  preserved 
by  the  numerical  composition  rule.  Conjoint  analysis  can 
be  defined  as  the  procedure  whereby  the  numerical  scale  values 
for  the  joint  effects  and  the  levels  of  the  independent  variables 
are  obtained.  One  attempts  to  find  the  appropriate  combination 
rule  and,  assuming  the  rule  is  valid,  finds  numerical  functions 
that  best  fit  the  order  of  the  joint  effects  in  the  data 
according  to  the  specified  rule. 

The  following  discussion  primarily  concerns  additive 
conjoint  measurement  in  two  factors.  Luce  and  Tukey  (1964) 
showed  that  given: 

(1)  the  set  A  =  A]_  x  A2,  where  A]_  and  A2  are  non-empty 
sets. 
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(2)  a  binary  relation  =£  on  A^  x  A2  (where  "2"  might 
be  a  relation  such  as  "is  preferred  over",  and 

(3)  if  <A^,  A2,£>  is  an  empirical  relational  system 
that  satisfies  the  axiom  structure  of  a  weak 
order  (i.e.,  ~  is  connected  and  transitive), 
solvability,  cancellation,  and  the  Arc  tiidean 
property,  then, 

THEOREM:  There  exist  real-valued  functions  +  n 

A,  4>i  on  A]_f  and  <J>2  on  A2  such  that  for  all  (a]_, 
a2)  and  (b^,  b2 )  in  A 

(i)  (ai,  a2 )  >  (bi,  b2 )  if  <j)(ai,  a2)>4>(*>i»  b2 ) , 


is  the  multiplicative  model 

4>  ( a  i  r  a2)  =  4>1  ( al )  4>2  ^  a2  ^ 

and  for  three  or  more  factors  there  is  a  distributive  model 
and  a  dual-distributive  model. 

Krantz  and  Tversky  (1971)  determined  ordinal  properties 
that  are  sufficient  for  each  of  the  four  basic  models  to 
hold.  A  necessary  condition  for  the  existence  of  an  additive 
model  for  two  factors  is  the  independence  of  and  A2  (that 
is,  A^  is  independent  of  A2  and  A2  is  independent  of  A^ ) . 

Ai  is  said  to  be  independent  of  A2  whenever 

(ai,  a2)  £  (b]_,  a2 )  if  and  only  if 
(ai,  b2 )  >  (b]_,  b2> 

Thus,  the  independence  of  A^  implies  that  if  a^  >  bj  for 
one  level  in  A2,  then  this  relation  wixl  hold  for  any  other 
level  in  A2. 

Given  the  set  of  necessary  ordinal  properties,  it 
is  possible  to  evaluate  each  of  the  four  basic  models  for 
a  set  of  observations  obtained  from  a  factorial  design. 

It  is  sufficient  to  require  each  subject  to  present  only 
rank-order  estimates  to  each  of  the  stimulus  combinations 
generated  by  combining  levels  of  the  factors.  In  most  applications 
of  conjoint  measurement,  theory,  it  is  the  additive  model 
that  is  of  interest.  However,  even  for  an  additive  model 
with  two  factors  of  several  levels,  both  the  testing  procedures 
for  the  ordinal  properties  and  the  scaling  procedure  for 
obtaining  the  numerical  scale  values  become  tedious  if  not 
impractical  without  the  aid  of  a  computer-based  algorithm. 


CONJOINT  MEASUREMENT  ALGORITHM 

There  are  computer  algorithms  which  test  whether 
the  necessary  ordinal  properties  hold.  Two  primary  programs 
are  CONJOINT  (Holt  and  Wallsten,  1974)  and  PCJM2  (Ullrich 
and  Cumming,  1973).  These  programs  test  for  independence 
among  the  factors  and  count  the  number  of  violations  of  the 
necessary  ordinal  properties,  specifying  the  data  cells  involved 
in  violations. 

There  are,  in  addition,  computer  algorithms  which 
use  nonmetric  conjoint  scaling  techniques  to  fit  a  given 
data  structure  to  an  additive,  multiplicative,  distributive 
or  dual-distributive  model.  Two  conjoint  scaling  approaches 
considered  in  the  present  research  are  the  delta-scaling 
method  (see  COOMBS,  1970)  and  the  computer  algorithm  MONANOVA 
(MONotonic  ANalysis  of  VAriance)  developed  by  Kruskal  (1965). 

The  delta  scaling  method  is  an  algorithm  for  converting 
an  ordinal  scale  to  a  scale  with  interval  properties  satisfying 
the  conditions  of  additive  conjoint  measurement.  In  this 
approach,  an  additive  representation  is  sought  which  satisfies 
inequalities  specified  by  the  orderings  of  preference  of 
the  elements  in  the  set  A  =  A^  x  A£.  The  solution  is  found 
using  linear  programming  methods.  Scale  values  for  levels 
of  the  factors  are  found.  If  the  number  of  factors  and  the 
levels  are  not  large,  this  approach  is  ammenable  to  "hand" 
calculation . 
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MONANOVA  is  used  to  fit  data  to  an  additive  model 


by  way  of  a  nonmetric  scaling  procedure.  In  MONANOVA,  a 
monotonic  transformation  of  the  data  is  found  which  leads 
to  estimates  of  the  scale  values  for  levels  of  the  factors. 
These  scale  values  are  such  that  when  they  are  combined  by 
way  of  an  additive  composition  rule,  their  joint  effects 
best  fit  the  original  data.  This  process  is  now  described. 
First,  an  initial  configuration  (an  initial  set  of  scale 
values  for  the  levels  of  the  factors)  is  generated  or  read 
by  the  program.  After  this  configuration  is  normalized, 
a  set  of  distances,  dj,  is  calculated  based  upon  the  additive 
rule.  The  program  next  uses  a  monotone  regression  subroutine 

A 

to  create  disparities,  dj,  which  are  modified  distances  subject 
to  the  constraint  that  they  are  monotonically  ordered  in 
the  same  way  as  their  corresponding  raw  data  values.  To 
estimate  how  well  the  model  fits  the  data,  a  goodness-of-f it 
measure  called  STRESS  is  computed.  STRESS  is  defined  by 


STRESS 


where  n  is  the  number  of  stimuli  and  d  is  the  mean  distance 
value.  If  the  model  were  to  fit  the  data  perfectly,  than 
the  STRESS  would  be  zero.  MONANOVA  uses  the  method  of  gradient 
in  an  iterative  fashion  to  find  the  configuration  that  will 


PROTOTYPE  EXAMPLE 


Conojoint  measurement  theory  is  examined  through 
a  prototype  example. 

A  new  fighter  aircraft  has  been  developed.  We  would 
like  to  quantify  a  test  pilot's  subjective  ratings  in  order 
to  compare  this  aircraft  with  other  classes  of  fighter  aircraft 
and  to  identify  strengths  and  weaknesses. 

Suppose  that  the  aircraft  is  to  be  subjectively  rated 
on  two  different  factors:  factor  A  and  factor  B.  Factor 
A  might  be  workload  and  factor  B  might  be  subsystem  effectiveness 
Let  factor  A  have  verbal  descriptive  levels  a^,  &2'  and 
33  and  factor  B  have  verbal  descriptive  levels  b^,  b2,  and 


A  MULTIFACTOR  ORDINAL  SCALE 


-j 

a 


Our  objective  is  to  convert  a  subjective  rating  pair 
-  a  rating  on  factor  A  and  a  rating  on  factor  B  -  into  an 
ordinal  number  that  represents  the  overall  quality  of  the 
aircraft.  We  will  develop  an  ordinal  scale  of  aircraft  quality. 

Four  basic  steps  are  required  to  convert  subjective 
ratings  into  meaningful  ordinal  numbers  so  that  a  comparison 
between  aircraft  can  be  made. 

First,  a  rating  standard  must  be  established  by  a 
group  of  pilots  to  which  the  test  pilot  belongs.  Each  pilot 
in  the  group  is  asked  to  rank  order  the  cells  (a^,  bj)  in 
the  rating  matrix  for  factors  A  and  B  in  Fig.  1,  using  the 
integers  1  through  9.  (The  row-column  arrangement  in  the 
rating  matrix,  i  is  the  row  and  j  is  the  column,  is  for  mathematical 
convenience.)  Suppose,  for  the  purpose  of  discussion,  that 
all  pilots  in  the  group  decide  upon  the  rank  ordering  given 
by  the  matrix  M 


M  = 


1 

2 

4 


3 

5 

7 


6 

8 

9 


(1) 


Thus,  (a3,  b3)  is  the  most  preferred  and  (a^,  b^ )  is  the 
least  preferred.  It  is  noted  that  rank  ordering  is  a  weak 
mathematical  description;  while  (33,  b3 )  is  preferred  over 
( a2 ,  b3>,  it  is  not  known  by  how  much.  Observe  that  in  M 
the  pilot  is  expressing  a  preference  in  which  he  weighs  the 


1 


N 


-I 

I 

5 


si 

N- 


3 

I 


e 
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importance  of  one  factor  with  another.  Matrix  M  defines 
a  transformation  in  which  each  verbal  descriptive  pair  (a^  b j ) 
is  mapped  to  a  unique  ordinal  number;  it  produces  an  ordinal 
scale  of  aircraft  quality. 

Second,  after  having  flown  the  new  aircraft,  the 
test  pilot  is  asked  to  give  subjective  ratings  for  factors 
A  and  B.  Suppose  his  ratings  are  those  indicated  by  x  in 
Fig.  2. 

Third,  the  test  pilot’s  subjective  ratings  are  converted 
to  a  single  ordinal  number  using  the  transformation  determined 
in  the  first  step.  Thus,  by  the  transformation  M  the  aircraft 
quality  is  an  8. 

Fourth,  the  ordinal  number  must  be  interpreted.  This 
is  simpler  than  in  an  independent  factor  by  factor  comparison. 
The  new  aircraft  is  "better"  than  another  fighter  aircraft 
with  a  7,  but  "inferior"  to  one  with  a  9. 

There  are  a  number  of  disadvantages  in  this  approach. 
First,  it  may  be  difficult  to  design  a  questionnaire  such 
that  there  is  general  agreement  in  the  pilot  group  on  the 
rank  ordering.  Second,  it  may  not  be  possible  for  the  human 
to  intelligently  rank  order  several  factors,  each  with  a 
number  of  levels.  Third,  an  aircraft  may  have  a  lower  ordinal 
rating  than  another,  but  it  is  not  immediately  known  which 
factor  contributed  most  to  the  problem.  Fourth,  the  ordinal 
numbers  provide  only  a  qualitative  measure. 

However,  the  composite  multifactor  approach,  provided 


the  questionnaire  is  properly  designed  and  the  transformation 
is  generally  agreed  upon,  can  be  expected  to  give  a  meaningful 
ordinal  number  by  which  aircraft  can  be  simply  and  directly 
compared. 


A  MULTIFACTOR  CONJOINT  MEASUREMENT  INTERVAL  SCALE 

We  would  like  to  convert  the  multifactor  ordinal 
scale  on  factors  A  and  B  into  numbers  that  have  greater  preciseness; 
we  will  use  conjoint  measurement  theory  to  develop  a  scale 
of  aircraft  quality  that  has  interval  properties.  In  this 
analysis  we  will,  in  addition,  prescribe  numbers  to  verbal 
descriptions  (that  is,  scale  these  descriptions)  and  model 
the  aircraft  quality  as  functions  of  the  given  factors. 

Conjoint  measurement  theory  has  a  subtle  relationship 
with  the  analysis  of  variance.  We  begin  by  examining  the 
underlying  model  for  the  analysis  of  variance. 

We  will  assume  in  our  initial  analysis  that  the  integers 
assigned  to  the  cells  of  the  rating  matrix  have  precise  mathematical 
meaning,  rather  than  only  expressing  an  ordering.  Thus, 
the  matrix  M  in  (1)  can  be  considered  as  data  from  a  two-factor 
experiment. 

We  consider  the  underlying  (deterministic)  model 
for  an  analysis  of  variance  of  a  two-factor  experiment  with 
three  levels  for  each  factor,  no  replication  effects,  and 
no  random  effects  in  the  data.  This  model  is  given  by 
y  ij  =  y  +  cti  +  Sj  +  (aB)ijf  (2) 

where  yj_j  is  the  observation  taken  at  the  ith  level  of  factor 
A  and  the  jth  level  of  factor  B,  y  is  the  grand  mean,  oti 
is  the  effect  of  the  ith  level  of  factor  A,  gj  is  the  effect 
of  the  jth  level  of  factor  B  and  (ag)ij  is  the  interaction 
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of  the  ith  level  of  factor  A  and  the  jth  level  of  factor 
B.  The  model  is  further  specified  by 

3  3  3  3 

Zdi  =  Z8j  =  E ( a6 ) i j  =  E(a8)ij  =  0.  (3) 

i=l  j=l  i=l  j=l 

We  identify  y^j  with  the  elements  of  M  and  solve  for  the 
parameters  in  (2).  The  relationships  in  (2)  and  (3)  provide 
seventeen  equations  that  uniquely  define  the  sixteen  parameters. 
The  parameters  are 

y  =  5 


(alf 

»2  • 

0.3)  = 

(-5/3,  0, 

5/3) 

b2- 

-  63)  = 

(-8/3,  0, 

8/3) 

'  1/3 

-1/3 

0 

otB )  i  j 

= 

-1/3 

0 

1/3 

.  0 

1/3 

-1/3 

We  make  two  observations.  First,  according  to  the 
model  there  are  interactive  effects.  Second,  the  model  has 
assigned  numbers  and  Bj  to  the  verbal  descriptive  levels 
a^  and  bj.  These  numbers  give  an  indication  of  how  "close" 
of  how  "far  apart"  the  verbal  descriptive  levels  are. 

We  illustrate  what  the  model  does.  Suppose  the  interactive 
effects  are  identically  zero,  then  the  data  would  be  represented 
by 

yij  =  y  +  +  Bj.  (5) 

This  equation  is  the  discrete  version  of  the  equation  of 
a  plane 


y  =  y  +  a  +  S 


(6) 


defined  on  the  a  and  6  axes.  Thus,  in  the  absence  of  interactive 
effects,  the  model  assigns  numerical  values  to  the  levels 
of  the  factors  A  and  B  in  such  a  way  that  the  data  values 
lie  in  the  plane  defined  by  (6).  If  there  are  interactive 
effects,  then  not  all  of  the  data  values  can  be  placed  in 
this  plane. 


Suppose  we  drop  the  interactive  terms  in  our  representation 

I 

of  M  in  (2).  We  then  have  a  new  set  of  numbers  y^j,  which 
in  general  are  different  from  the  data  values  yij»  defined 
by 


*< 

i_i. 

■■ 

=  U  + 

ai  +  8  j , 

• 

(7) 

with 

Ur 

«i 

and  Bj  given  in  (4). 

We  introduce  the 

variables 

1 

ai  * 

«i  + 

5/3, 

(8) 

8-  = 
3 

8  j  + 

8/3, 

(9) 

If 

y ij  s 

8  yij 

-  2/3, 

(10) 

into 

(7) 

to 

obtain 

If 

Yij  s 

=  ai  + 

■  6  j  • 

(11) 

We  denote 

the  set 

of  values  yij 

by  T.  Matrix  T  is 

given 

by 

0 

2.67 

5.33 

T  = 

1.67 

4.33 

7.00 

(12) 

3.33 

6.00 

8.67 

(The 

elements  of  r. 

C  are 

fractions 

;  however,  for  the 

purpose 

of  later  comparisons,  they  are  expressed  as  decimals.)  Observe 
that  while  the  matrix  elements  in  T  have  different  values 
than  those  in  M,  the  rank  ordering  of  the  elements  has  been 
preserved. 


We  have  in  effect  transformed  the  data  in  M  to  "new 
data"  in  T  in  such  a  way  that  the  rank  ordering  is  preserved 
and  the  new  data  satisfies  an  additive  model.  This  transformation 
of  data  is  illustrated  in  Pig.  3.  Note  that  numbers  have 
been  assigned  to  the  verbal  descriptive  levels  (that  is, 
they  have  been  scaled)  and  each  cell  value  in  the  new  data 
matrix  is  the  sum  of  the  row  and  column  scales. 

What  we  have  done  is  the  essence  of  conjoint  measurement; 
we  have  found  a  transformation  of  the  data  that  preserves 
the  rank  ordering  and  such  that  a  model  (in  this  case,  an 
additive  model)  holds.  Through  the  transformation  we  have 
produced  a  scale  of  aircraft  quality  that  has  interval  properties 
(the  elements  of  T  have  precise  numerical  values  while  the 
elements  of  M,  as  originally  defined,  express  only  a  rank 
ordering).  We  have  also  scaled  the  verbal  descriptive  levels 
of  each  factor.  The  model  provides  a  simple  law:  the  measure 
of  aircraft  quality  is  the  sum  of  two  scaled  factors. 

We  transformed  the  data  by  neglecting  the  interaction 
terms  in  the  deterministic  model  for  the  analysis  of  variance. 

The  general  approach  in  additive  conjoint  measurement,  however, 
is  to  systematically  search  for  the  order  preserving  transformation 
such  that  the  interactive  effects  are  best  reduced  in  the 
transformed  data.  Two  conjoint  measurement  methods  are  now 
examined:  delta-scaling  and  MONANOVA. 
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Fig.  3. 


"Old  Data"  Transformed  into  "New  Scaled  Data" 


AN  INTERVAL  SCALE  DERIVED  BY  DELTA  SCALING 


In  the  previous  section  we  used  the  underlying  model 
for  the  analysis  of  variance  to  transform  the  ordinal  scale 
to  an  interval  scale  in  such  a  way  that  an  additive  model 
held.  We  now  make  connections  to  the  earlier  presentation 
of  conjoint  measurement  theory. 

The  assignment  of  preference  to  the  cells  (ai,  bj) 
for  factors  A  and  B,  given  in  the  matrix  M  in  (1),  satisfies 
the  axioms  in  the  conjoint  measurement  theorem  for  an  additive 
model.  In  particular,  the  factors  A  and  B  are  independent. 

(The  rows  (columns)  in  M  are  all  of  the  same  rank  order.) 

Thus,  by  the  theorem,  there  exist  real-valued  functions  <p 
on  A  x  B,  <J>i  on  A,  and  <f>2  on  B  such  that  for  all  (a^,  bj) 
in  A,  <p(aj_,  bj)  =  <j>y(ai)  +  <J>2  ( b  j ) .  Furthermore,  any  other 
functions  <J>',  and  $2  which  satisfy  the  theorem  are  related 

through  linear  transformations  to  <J>,  <j>y,  and  <j>2« 

The  basic  theorem  does  not  provide  a  procedure  to 
obtain  the  functions  <J>,  and  4> 2 -  However,  in  the  previous 

section,  we  in  effect  constructed  a  set  of  functions  that 
had  the  additive  property.  Thus,  any  other  set  of  functions 
which  we  might  determine  by  other  methods  must  be  related 
to  the  functions  we  have  already  found. 

We  now  apply  the  delta  scaling  method  to  our  subjective 
rating  problem.  The  delta  scaling  method  has  been  used  in 
the  assessment  of  fighter  aircraft  (Helm  et.al.,  1974).  We 


use  the  "tally  sheet"  procedure  in  the  cited  reference,  set 


all  "d"s  equal  to  unity,  and  obtain  the  matrix  T' 


T' 


0.  7.  14. 

5.  12.  19. 

9.  16.  23. 


(13) 


As  in  T,  each  element  in  T'  is  the  sum  of  the  corresponding 
row  and  column  scales.  Each  element  in  T'  is  a  constant 
multiple  of  the  corresponding  element  in  T,  as  guaranteed 
by  the  theorem. 

We  have  through  conjoint  measurement  theory  converted 
an  ordinal  scale  to  an  interval  scale  in  such  a  way  that 
an  additive  model  holds. 

Delta  scaling  by  hand  computation  is  generally  impractical 
In  further  analysis  the  computer  algorithm  MONANOVA  is  used. 


AN  INTERVAL  SCALE  DERIVED  BY  MONANOVA 


MONANOVA  is  an  algorithm  that  seeks  the  monotone 
(hence,  order  preserving)  transformation  of  the  data  such 
that  the  transformed  data  best  fit  an  assumed  linear  model. 

It  employs  a  gradient  search.  The  degree  to  which  the  transformed 
data  fit  the  model  is  indicated  by  the  STRESS. 

We  apply  MONANOVA  to  the  ordinal  matrix  defined  by 
M  in  (1).  For  an  additive  model,  the  MONANOVA  computer  algorithm 
produces,  with  zero  STRESS,  the  matrix  T"  defined  by 


o 

o 

• 

1.47 

2.94 

.92 

2.39 

3.86 

1.84 

3.32 

4.77 

Scales  for  factor  A  and  factor  B  are  given  in  T"  by  the  first 
column  and  first  row,  respectively.  Each  element  in  T"  is 
the  sum  of  the  corresponding  row  and  column  scales. 

As  in  T'  in  (13),  each  element  in  T"  is  a  constant 


multiple  of  the  corresponding  element  in  T. 


SENSITIVITY  ANALYSIS 


Through  conjoint  measurement  we  have  converted  an 
ordinal  scale  of  aircraft  quality  to  one  that  has  interval 
properties.  We  will  consider  how  the  interval  scale  values 
change  when  there  are  small  perturbations  in  the  rank  orderings 
We  use  as  the  unperturbed  state  the  rank  ordering 


in  Mj. 


(15) 


and  as  perturbed  states 

1  3 

I 

Mi  = 


and  Mi 
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(16) 


(17) 


Matrices  m{  and  Mi  are  obtained  from  Mi  by  interchanging 
two  elements.  The  perturbed  rank  orderings  are  reasonable. 

So  that  we  can  make  comparisons,  we  "normalize"  the 
computed  interval  scale  so  that  the  largest  interval  value 
is  100. 


One  would  hope  that  the  perturbations  in  the  rank 
orderings  would  cause  only  local  effects  in  the  interval 


scale  values.  But  this  is  not  the  case,  as  may  be  seen  m 
Fig.  4.  The  perturbation  given  by  M{  caused  a  shift  of  the 
interval  scale  values  to  the  right  and  the  perturbation  given 
by  caused  a  shift  of  the  interval  scale  values  to  the 
left. 

These  general  shifts  make  it  difficult  to  interpret 
the  interval  scale  values  and  cast  doubt  upon  the  validity 
of  the  scales.  Even  though  the  cell  with  the  ordinal  8  was 
not  perturbed,  it  was  mapped  through  conjoint  measurement 
to  80.9,  84.6  and  73.1  for  Mi,  m{  and  Mi ,  respectively. 

Suppose  it  was  believed  that  75.  was  a  "passing"  score. 

Then  an  8  or  9  would  be  passing  if  Mi  was  the  standard;  a 
7,  8,  or  9  would  be  passing  if  Mi  was  the  standard;  and  only 
9  would  be  passing  if  Mi  was  the  standard. 

We  examine  the  perturbation  effects  when  the  number 
of  levels  is  increased.  We  assume  four  levels  for  each  factor 

We  define  the  unperturbed  state  by  M2 
"l  3  6  10 

M2  =  2  5  9  13  (18) 

4  8  12  15 

7  11  14  16_ 

And  the  perturbed  states  by  M2 

fl  3  6  9l 
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The  interval  scale  values  determined  through  conjoint 
measurement  for  the  ordinal  data  in  M2 ,  and  are  given 

in  Fig.  5.  While  the  perturbations  still  cause  shifts  in 
the  interval  scale  values,  they  are  not  as  pronounced  as 
in  the  three-level  case. 


SUMMARY 


A  subjective  rating  pair  on  factors  A  and  B  in  the 
prototype  example  was  converted  into  an  ordinal  number  that 
represented  the  overall  aircraft  quality  using  four  basic 
steps.  First,  a  rating  standard  was  established  in  which 
the  factor  pairs  were  rank  ordered.  Second,  the  pilot  was 
asked  to  rate  the  new  aircraft  in  the  two  factors.  Third, 
the  pilot's  subjective  ratings  were  converted  into  a  single 
ordinal  number  using  the  transformation  of  the  first  step. 

Fourth,  the  ordinal  number  was  interpreted,  allowing  comparisons 
between  aircraft. 

There  are  a  number  of  disadvantages  of  this  approach. 
First,  there  may  not  be  general  agreement  in  the  pilot  group 
on  the  rank  ordering.  Second,  the  human  is  limited  in  his 
capability  to  rank  order  several  factors,  each  with  a  number 
of  levels.  Third,  the  role  of  a  specific  factor  is  disguised 
in  the  ordinal  scale.  Fourth,  the  ordinal  numbers  provide 
only  a  qualitative  measure. 

If,  however,  the  questionnaire  is  properly  designed 
and  the  rank  ordering  is  generally  agreed  upon,  this  approach 
can  be  expected  to  produce  a  meaningful  ordinal  number  by 
which  aircraft  can  be  simply  and  directly  compared. 

Conjoint  measurement  theory  was  introduced  using 
the  underlying  model  for  the  analysis  of  variance  in  a  two-factor 
experiment.  When  the  interactive  terms  in  this  model  were 


dropped,  the  model  produced  an  interval  scale  with  additive 
properties.  The  verbal  descriptive  levels  were  scaled;  that 
is,  numbers  were  assigned  to  them. 

The  general  approach  in  additive  conjoint  measurement 
theory  is  to  systematically  search  for  the  order  preserving 
transformation  such  that  the  interactive  effects  are  best 
reduced  in  the  transformed  data.  Two  conjoint  measurement 
methods  were  examined:  delta  scaling  and  MONANOVA. 

The  delta-scaling  method  produced  an  interval  scale 
with  additive  properties.  The  associated  transformation 
differed  from  that  of  the  analysis  of  variance  model  by  a 
constant  factor  as  guaranteed  by  the  conjoint  measurement 
theorem  for  an  additive  model. 

An  interval  scale  was  derived  using  the  computer 
algorithm  MONANOVA.  MONANOVA  is  an  algorithm  that  seeks 
the  monotone  transformation  of  data  such  that  the  transformed 
data  best  fit  an  assumed  linear  model.  The  associated  transformation 
again  differed  by  constant  factors  from  those  of  the  other 
two  approaches. 

A  sensitivity  analysis  was  made  using  MONANOVA. 

One  might  expect  that  "small"  perturbations  in  the  rank  orderings 
would  cause  only  local  effects  in  the  interval  scale  values. 

This  was  not  the  case,  for  when  two  elements  were  interchanged, 
large  shifts  occurred  in  the  interval  scales.  When  ♦■he  number 
of  levels  was  increased  from  three  to  four,  thv=  perturbation 
influence  on  the  interval  scale  was  reduced. 


CONCLUSIONS  AND  RECOMMENDATIONS 


The  multifactor  ordinal  scale  is  a  meaningful  measure 
of  aircraft  quality.  Like  all  subjective  ratings,  its  validity 
depends  upon  whether  there  is  a  generally  agreed  upon  rating 
standard  and  whether  the  test  pilot's  subjective  ratings 
conform  to  the  standard.  These  issues  can  be  largely  managed 
through  the  selection  and  training  of  test  pilots  and  through 
the  design  of  questionnaires. 

An  improvement  in  the  ordinal  scale  could  be  made 
by  allowing  a  more  quantitative  ordering;  for  example,  by 
allowing  equal  or  great-than-integer  ordering.  Also,  it 
may  be  appropriate  to  assign  scalar  values  to  the  cells  of 
the  rating  matrix. 

The  primary  limitation  of  the  multifactor  ordinal 
scale  is  the  ability  of  the  human  to  rank  order  several  factors 
with  several  levels.  It  may  be  possible  to  develop  mathematical 
models  to  assist  the  human  in  the  rank  ordering.  For  example, 
there  may  be  enough  information  to  determine  a  complete  rank 
ordering  if  the  human  rank  orders  the  levels  of  the  factors 
and  also  rank  orders  the  factors  themselves. 

The  interval  scale  determined  by  conjoint  measurement 
theory  does  not  appear  to  be  an  improvement  over  the  ordinal 
scale  in  the  prototype  example.  To  begin  with,  imprecise 
data  is  generated  through  the  imposition  of  the  weakness 
of  rank  ordering.  (The  assignment  of  scalar  values  to  the 


cells  of  the  rating  matrix  could  make  the  data  more  precise. ) 
This  data  in  effect  is  then  transformed  to  fit  a  model. 

This  approach  would  be  acceptable  if  the  model  reflected 
the  "true  world"  and  it  acted  to  "improve"  the  data.  However 
there  is  no  assurance  that  a  specific  model  will  improve 
the  data;  and  thus  the  interval  scales  have  questionable 
validity.  The  major  changes  in  the  interval  scales  caused 
by  small  perturbations  in  the  rating  matrix  illustrate  the 
difficulty  in  interpreting  the  interval  scales. 

Conjoint  measurement  methods  should  be  explored  as 
a  basic  modeling  and  scaling  approach  -  along  with  other 
methods  -  to  seek  the  constitutive  laws  of  subjective  ratings 
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